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Abstract 


Training Deep Neural Networks is complicated by the fact 
that the distribution of each layer”s inputs changes during 
training, as the parameters of the previous layers change. 
This slows down the training by requiring lower learning 
rates and careful parameter initialization, and makes it no- 
toriously hard to train models with saturating nonlineari- 
ties. We refer to this phenomenon as internal covariate 
shift, and address the problem by normalizing layer in- 
puts. Our method draws its strength from making normal- 
ization a part of the model architecture and performing the 
normalization for each training mini-batch. Batch Nor- 
malization allows us to use much higher learning rates and 
be less careful about initialization. It also acts as a regu- 
larizer, in some cases eliminating the need for Dropout. 
Applied to a state-of-the-art image classification model, 
Batch Normalization achieves the same accuracy with 14 
times fewer training steps, and beats the original model 
by a significant margin. Using an ensemble of batch- 
normalized networks, we improve upon the best published 
result on ImageNet classification: reaching 4.9% top-5 
validation error (and 4.8% test error), exceeding the ac- 
curacy of human raters. 


1 Introduction 


Deep learning has dramatically advanced the state of the 
art in vision, speech, and many other areas. Stochas- 
tic gradient descent (SGD) has proved to be an effec- 
tive way of training deep networks, and SGD variants 
such as momentum and Adagrad 
2011) have been used to achieve state of the 


art performance. SGD optimizes the parameters O of the 
network, so as to minimize the loss 


N 
O = arg min > Do L(x;, O) 

i=1 
where x1...y is the training data set. With SGD, the train- 
ing proceeds in steps, and at each step we consider a mini- 
batch X1...m of size m. The mini-batch is used to approx- 
imate the gradient of the loss function with respect to the 
parameters, by computing 


1 OL(xi, 0) 


m oO ` 
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Using mini-batches of examples, as opposed to one exam- 
ple at a time, is helpful in several ways. First, the gradient 
of the loss over a mini-batch is an estimate of the gradient 
over the training set, whose quality improves as the batch 
size increases. Second, computation over a batch can be 
much more efficient than m computations for individual 
examples, due to the parallelism afforded by the modern 
computing platforms. 

While stochastic gradient is simple and effective, it 
requires careful tuning of the model hyper-parameters, 
specifically the learning rate used in optimization, as well 
as the initial values for the model parameters. The train- 
ing is complicated by the fact that the inputs to each layer 
are affected by the parameters of all preceding layers — so 
that small changes to the network parameters amplify as 
the network becomes deeper. 

The change in the distributions of layers’ inputs 
presents a problem because the layers need to continu- 
ously adapt to the new distribution. When the input dis- 
tribution to a learning system changes, it is said to experi- 
ence covariate shift (Shimodaira,|2000). This is typically 
handled via domain adaptation (Jiang, 2008). However, 
the notion of covariate shift can be extended beyond the 
learning system as a whole, to apply to its parts, such as a 
sub-network or a layer. Consider a network computing 






l= Fa(F (u, O1), O2) 


where Fi and F> are arbitrary transformations, and the 
parameters ©1,©2 are to be learned so as to minimize 
the loss £. Learning O2 can be viewed as if the inputs 
x = F; (u, 01) are fed into the sub-network 


L= F2(x, Og). 


For example, a gradient descent step 


a Ss OF2(xi, O2) 
O) + O) — m2 de 


(for batch size m and learning rate a) is exactly equivalent 
to that for a stand-alone network F> with input x. There- 
fore, the input distribution properties that make training 
more efficient — such as having the same distribution be- 
tween the training and test data — apply to training the 
sub-network as well. As such it is advantageous for the 
distribution of x to remain fixed over time. Then, 02 does 
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not have to readjust to compensate for the change in the 
distribution of x. 


Fixed distribution of inputs to a sub-network would 
have positive consequences for the layers outside the sub- 
network, as well. Consider a layer with a sigmoid activa- 
tion function z = g(Wu + b) where u is the layer input, 
the weight matrix W and bias vector b are the layer pa- 
rameters to be learned, and g(x) = oe As |x| 
increases, g'(x) tends to zero. This means that for all di- 
mensions of x = Wu-+-b except those with small absolute 
values, the gradient flowing down to u will vanish and the 
model will train slowly. However, since x is affected by 
W, b and the parameters of all the layers below, changes 
to those parameters during training will likely move many 
dimensions of x into the saturated regime of the nonlin- 
earity and slow down the convergence. This effect is 
amplified as the network depth increases. In practice, 
the saturation problem and the resulting vanishing gradi- 
ents are usually addressed by using Rectified Linear Units 
(Nair & Hinton x) = max(z,0), careful 
initialization (Bengio & Glorot, (2013), 
and small learning rates. If, however, we could ensure 
that the distribution of nonlinearity inputs remains more 
stable as the network trains, then the optimizer would be 
less likely to get stuck in the saturated regime, and the 
training would accelerate. 












We refer to the change in the distributions of internal 
nodes of a deep network, in the course of training, as In- 
ternal Covariate Shift. Eliminating it offers a promise of 
faster training. We propose a new mechanism, which we 
call Batch Normalization, that takes a step towards re- 
ducing internal covariate shift, and in doing so dramati- 
cally accelerates the training of deep neural nets. It ac- 
complishes this via a normalization step that fixes the 
means and variances of layer inputs. Batch Normalization 
also has a beneficial effect on the gradient flow through 
the network, by reducing the dependence of gradients 
on the scale of the parameters or of their initial values. 
This allows us to use much higher learning rates with- 
out the risk of divergence. Furthermore, batch normal- 
ization regularizes the model and reduces the need for 
Dropout 12014). Finally, Batch Normal- 
ization makes it possible to use saturating nonlinearities 
by preventing the network from getting stuck in the satu- 
rated modes. 


In Sec. we apply Batch Normalization to the best- 
performing ImageNet classification network, and show 
that we can match its performance using only 7% of the 
training steps, and can further exceed its accuracy by a 
substantial margin. Using an ensemble of such networks 
trained with Batch Normalization, we achieve the top-5 
error rate that improves upon the best known results on 
ImageNet classification. 
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2 Towards Reducing Internal 


Covariate Shift 


We define Internal Covariate Shift as the change in the 
distribution of network activations due to the change in 
network parameters during training. To improve the train- 
ing, we seek to reduce the internal covariate shift. By 
fixing the distribution of the layer inputs x as the training 
progresses, we expect to improve the training speed. It has 
been long known 
that the network training converges faster if its in- 
puts are whitened — i.e., linearly transformed to have zero 
means and unit variances, and decorrelated. As each layer 
observes the inputs produced by the layers below, it would 
be advantageous to achieve the same whitening of the in- 
puts of each layer. By whitening the inputs to each layer, 
we would take a step towards achieving the fixed distri- 
butions of inputs that would remove the ill effects of the 
internal covariate shift. 


We could consider whitening activations at every train- 
ing step or at some interval, either by modifying the 
network directly or by changing the parameters of the 
optimization algorithm to depend on the network ac- 


tivation values (Wiesler et al. 2014 Raiko et al | 2012 
Desjardins & Kavukcuoglu). How- 


ever, if these modifications are interspersed with the op- 
timization steps, then the gradient descent step may at- 
tempt to update the parameters in a way that requires 
the normalization to be updated, which reduces the ef- 
fect of the gradient step. For example, consider a layer 
with the input u that adds the learned bias b, and normal- 
izes the result by subtracting the mean of the activation 
computed over the training data: = x — Elx] where 
x=u+b, Y = {x_n} is the set of values of x over 
the training set, and Elx] = + > zi. If a gradient 
descent step ignores the dependence of Elx] on b, then it 
will update b < b + Ab, where Ab x —0l/0%. Then 
u + (b+ Ab) — Elu + (b + Ab)] = u + b — Efu + b]. 
Thus, the combination of the update to b and subsequent 
change in normalization led to no change in the output 
of the layer nor, consequently, the loss. As the training 
continues, b will grow indefinitely while the loss remains 
fixed. This problem can get worse if the normalization not 
only centers but also scales the activations. We have ob- 
served this empirically in initial experiments, where the 
model blows up when the normalization parameters are 
computed outside the gradient descent step. 






O 









The issue with the above approach is that the gradient 
descent optimization does not take into account the fact 
that the normalization takes place. To address this issue, 
we would like to ensure that, for any parameter values, 
the network always produces activations with the desired 
distribution. Doing so would allow the gradient of the 
loss with respect to the model parameters to account for 
the normalization, and for its dependence on the model 
parameters O. Let again x be a layer input, treated as a 
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vector, and Y be the set of these inputs over the training 
data set. The normalization can then be written as a trans- 
formation 

x = Norm(x, 4) 


which depends not only on the given training example x 
but on all examples Y — each of which depends on O if 
x is generated by another layer. For backpropagation, we 
would need to compute the Jacobians 


ONorm(x, X) 
——_——— an 
Ox 


ignoring the latter term would lead to the explosion de- 
scribed above. Within this framework, whitening the layer 
inputs is expensive, as it requires computing the covari- 
ance matrix Cov[x] = Exexv[xx?] — E[xJE[x]7 and its 
inverse square root, to produce the whitened activations 
Cov|x]~!/?(x — E[x]), as well as the derivatives of these 
transforms for backpropagation. This motivates us to seek 
an alternative that performs input normalization in a way 
that is differentiable and does not require the analysis of 
the entire training set after every parameter update. 

Some of the previous approaches (e.g. 
2008)) use statistics computed 
over a single training example, or, in the case of image 
networks, over different feature maps at a given location. 
However, this changes the representation ability of a 
network by discarding the absolute scale of activations. 
We want to a preserve the information in the network, by 
normalizing the activations in a training example relative 
to the statistics of the entire training data. 


d ONorm(x, Y) 
OX 


3 Normalization via Mini-Batch 
Statistics 


Since the full whitening of each layer’s inputs is costly 
and not everywhere differentiable, we make two neces- 
sary simplifications. The first is that instead of whitening 
the features in layer inputs and outputs jointly, we will 
normalize each scalar feature independently, by making it 
have the mean of zero and the variance of 1. For a layer 
with d-dimensional input x = (2) ...a(D), we will nor- 
malize each dimension 


¿0 — rE — Ela] 
Var[a(*)] 


where the expectation and variance are computed over the 
training data set. As shown in (LeCun et al. [1998b), such 
normalization speeds up convergence, even when the fea- 
tures are not decorrelated. 

Note that simply normalizing each input of a layer may 
change what the layer can represent. For instance, nor- 
malizing the inputs of a sigmoid would constrain them to 
the linear regime of the nonlinearity. To address this, we 
make sure that the transformation inserted in the network 
can represent the identity transform. To accomplish this, 





we introduce, for each activation x), a pair of parameters 
y (4), 8%), which scale and shift the normalized value: 


¿O 030 y gl), 


These parameters are learned along with the original 
model parameters, and restore the representation power 
of the network. Indeed, by setting y(*) = \/Var[z™)] and 
BW) = Elx(*], we could recover the original activations, 
if that were the optimal thing to do. 

In the batch setting where each training step is based on 
the entire training set, we would use the whole set to nor- 
malize activations. However, this is impractical when us- 
ing stochastic optimization. Therefore, we make the sec- 
ond simplification: since we use mini-batches in stochas- 
tic gradient training, each mini-batch produces estimates 
of the mean and variance of each activation. This way, the 
statistics used for normalization can fully participate in 
the gradient backpropagation. Note that the use of mini- 
batches is enabled by computation of per-dimension vari- 
ances rather than joint covariances; in the joint case, reg- 
ularization would be required since the mini-batch size is 
likely to be smaller than the number of activations being 
whitened, resulting in singular covariance matrices. 

Consider a mini-batch B of size m. Since the normal- 
ization is applied to each activation independently, let us 
focus on a particular activation x") and omit k for clarity. 
We have m values of this activation in the mini-batch, 


Let the normalized values be %1._, and their linear trans- 
formations be y1...m. We refer to the transform 


BN, 6 | Lim >? Yl..m 


as the Batch Normalizing Transform. We present the BN 
Transform in Algorithm[]] In the algorithm, € is a constant 
added to the mini-batch variance for numerical stability. 


Input: Values of x over a mini-batch: B = {21,m}; 
Parameters to be learned: y, 8 
Output: {yi = BN, (a;)} 


1 m 
HB © m 2, 
= 


// mini-batch mean 


// mini-batch variance 


// normalize 


Yi E Yi + B = BN} g(x) // scale and shift 


Algorithm 1: Batch Normalizing Transform, applied to 
activation x over a mini-batch. 


The BN transform can be added to a network to manip- 
ulate any activation. In the notation y = BN, g(x), we 
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indicate that the parameters y and ĝ are to be learned, 
but it should be noted that the BN transform does not 
independently process the activation in each training ex- 
ample. Rather, BN} g(x) depends both on the training 
example and the other examples in the mini-batch. The 
scaled and shifted values y are passed to other network 
layers. The normalized activations 7 are internal to our 
transformation, but their presence is crucial. The distri- 
butions of values of any T has the expected value of 0 
and the variance of 1, as long as the elements of each 
mini-batch are sampled from the same distribution, and 
if we neglect e. This can be seen by observing that 
XiTi = 0 and + SO", 7? = 1, and taking expec- 
tations. Each normalized activation 5(*) can be viewed as 
an input to a sub-network composed of the linear trans- 
form y) = yZ) + 80), followed by the other pro- 
cessing done by the original network. These sub-network 
inputs all have fixed means and variances, and although 
the joint distribution of these normalized 2% can change 
over the course of training, we expect that the introduc- 
tion of normalized inputs accelerates the training of the 
sub-network and, consequently, the network as a whole. 

During training we need to backpropagate the gradi- 
ent of loss £ through this transformation, as well as com- 
pute the gradients with respect to the parameters of the 
BN transform. We use chain rule, as follows (before sim- 
plification): 


ol _ ol 

















oa; Oy 

OL. _ m OL —1 2 —3/2 
voz 7 oi da” (zi — ua): (op + €) / 

Ol _ m ae —1 ae ¿1 722 HB) 
dan = Ge BE; +) tag m 
bb — L. l q 32. Ari 8) y Oe 1 

Ozi Ti [oh +e 00% m Ong m 

ae _ ym A a 

Hawn, La 

Ol _ ym 3L 

BB = Limi By, 


Thus, BN transform is a differentiable transformation that 
introduces normalized activations into the network. This 
ensures that as the model is training, layers can continue 
learning on input distributions that exhibit less internal co- 
variate shift, thus accelerating the training. Furthermore, 
the learned affine transform applied to these normalized 
activations allows the BN transform to represent the iden- 
tity transformation and preserves the network capacity. 


3.1 Training and Inference with Batch- 
Normalized Networks 


To Batch-Normalize a network, we specify a subset of ac- 
tivations and insert the BN transform for each of them, 
according to Alg.|1} Any layer that previously received 
x as the input, now receives BN(x). A model employing 
Batch Normalization can be trained using batch gradient 
descent, or Stochastic Gradient Descent with a mini-batch 
size m > 1, or with any of its variants such as Adagrad 
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(Duchi et al. 2011). The normalization of activations that 


depends on the mini-batch allows efficient training, but is 
neither necessary nor desirable during inference; we want 
the output to depend only on the input, deterministically. 
For this, once the network has been trained, we use the 
normalization 
x x — Elx] 
@ = — SE 
Var[x] + e 
using the population, rather than mini-batch, statistics. 
Neglecting e, these normalized activations have the same 
mean 0 and variance | as during training. We use the un- 
biased variance estimate Var[x] = 2% - Eg[0%], where 
the expectation is over training mini-batches of size m and 
oR are their sample variances. Using moving averages in- 
stead, we can track the accuracy of a model as it trains. 
Since the means and variances are fixed during inference, 
the normalization is simply a linear transform applied to 
each activation. It may further be composed with the scal- 
ing by y and shift by 5, to yield a single linear transform 
that replaces BN(x). Algorithm[2]summarizes the proce- 
dure for training batch-normalized networks. 


Input: Network N with trainable parameters O; 
subset of activations {a VC, 
Output: Batch-normalized network for inference, 
1: Ngn & N // Training BN network 
2: fork =1...K do 
3: Add transformation y? = BN, 0) go (x) to 
tí, (Alg. 
Modify each layer in Ngy with input a”) to take 
y) instead 
: end for 
6: Train Ny to optimize the parameters O U 
{y), BALE 
: NBE < Ngy // Inference BN network with frozen 
// parameters 


: fork =1...k do 
/ For clarity, x= xP, y= 7), ug = po, etc. 
Process multiple training mini-batches 6, each of 
size m, and average over them: 
Elx] — Eslug] 
"Eglo 5] 


m—l1 


inf 
BN 


Var[x] — 
inf 
In NN, 


= — a . 
Y y Var[2]+e £ 


12: end for 


replace the transform y = BN} g(x) with 


— El] 
+ (8 rata 





Algorithm 2: Training a Batch-Normalized Network 


3.2 Batch-Normalized Convolutional Net- 
works 


Batch Normalization can be applied to any set of acti- 
vations in the network. Here, we focus on transforms 
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that consist of an affine transformation followed by an 
element-wise nonlinearity: 


z= g(Wu+b) 


where W and b are learned parameters of the model, and 
g(-) is the nonlinearity such as sigmoid or ReLU. This for- 
mulation covers both fully-connected and convolutional 
layers. We add the BN transform immediately before the 
nonlinearity, by normalizing x = Wu +b. We could have 
also normalized the layer inputs u, but since u is likely 
the output of another nonlinearity, the shape of its distri- 
bution is likely to change during training, and constraining 
its first and second moments would not eliminate the co- 
variate shift. In contrast, Wu + b is more likely to have 
a symmetric, non-sparse distribution, that is “more Gaus- 
sian” (Hyvärinen & Oja,/2000); normalizing it is likely to 
produce activations with a stable distribution. 

Note that, since we normalize Wu-+b, the bias b can be 
ignored since its effect will be canceled by the subsequent 
mean subtraction (the role of the bias is subsumed by in 
Alg.[T). Thus, z = g(Wu + b) is replaced with 


z = g(BN(Wu)) 


where the BN transform is applied independently to each 
dimension of x = Wu, with a separate pair of learned 
parameters 7"), 8%) per dimension. 

For convolutional layers, we additionally want the nor- 
malization to obey the convolutional property — so that 
different elements of the same feature map, at different 
locations, are normalized in the same way. To achieve 
this, we jointly normalize all the activations in a mini- 
batch, over all locations. In Alg.f we let B be the set of 
all values in a feature map across both the elements of a 
mini-batch and spatial locations — so for a mini-batch of 
size m and feature maps of size p x q, we use the effec- 
tive mini-batch of size m’ = |B| = m - pq. We learn a 
pair of parameters y“) and 6“) per feature map, rather 
than per activation. Alg.[2]is modified similarly, so that 
during inference the BN transform applies the same linear 
transformation to each activation in a given feature map. 


3.3 Batch Normalization enables higher 


learning rates 


In traditional deep networks, too-high learning rate may 
result in the gradients that explode or vanish, as well as 
getting stuck in poor local minima. Batch Normaliza- 
tion helps address these issues. By normalizing activa- 
tions throughout the network, it prevents small changes 
to the parameters from amplifying into larger and subop- 
timal changes in activations in gradients; for instance, it 
prevents the training from getting stuck in the saturated 
regimes of nonlinearities. 

Batch Normalization also makes training more resilient 
to the parameter scale. Normally, large learning rates may 
increase the scale of layer parameters, which then amplify 
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the gradient during backpropagation and lead to the model 
explosion. However, with Batch Normalization, back- 
propagation through a layer is unaffected by the scale of 
its parameters. Indeed, for a scalar a, 


BN(Wu) = BN((aW)u) 


and we can show that 


OBN((aW)u) _ OBN(Wu) 

du z ðu 
OBN((aW)u) _ 1 OBN(Wu) 
aw) a Ow 


The scale does not affect the layer Jacobian nor, con- 
sequently, the gradient propagation. Moreover, larger 
weights lead to smaller gradients, and Batch Normaliza- 
tion will stabilize the parameter growth. 

We further conjecture that Batch Normalization may 
lead the layer Jacobians to have singular values close to 1, 
which is known to be beneficial for training 
2013). Consider two consecutive layers with normalized 
inputs, and the transformation between these normalized 
vectors: Z = F(X). If we assume that X and Z are Gaussian 
and uncorrelated, and that F(X) ~ JX is a linear transfor- 
mation for the given model parameters, then both X and Z 
have unit covariances, and 7 = Covíz] = JCov{x|J7 
JJ", Thus, JJ? = I, and so all singular values of J 
are equal to 1, which preserves the gradient magnitudes 
during backpropagation. In reality, the transformation is 
not linear, and the normalized values are not guaranteed to 
be Gaussian nor independent, but we nevertheless expect 
Batch Normalization to help make gradient propagation 
better behaved. The precise effect of Batch Normaliza- 
tion on gradient propagation remains an area of further 
study. 


3.4 Batch Normalization regularizes the 
model 


When training with Batch Normalization, a training ex- 
ample is seen in conjunction with other examples in the 
mini-batch, and the training network no longer produc- 
ing deterministic values for a given training example. In 
our experiments, we found this effect to be advantageous 
to the generalization of the network. Whereas Dropout 
is typically used to reduce over- 
fitting, in a batch-normalized network we found that it can 
be either removed or reduced in strength. 


4 Experiments 


4.1 Activations over time 


To verify the effects of internal covariate shift on train- 
ing, and the ability of Batch Normalization to combat it, 


we considered the problem of predicting the digit class on 
the MNIST dataset (LeCun et al.,/1998a). We used a very 


simple network, with a 28x28 binary image as input, and 
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Figure 1: (a) The test accuracy of the MNIST network 
trained with and without Batch Normalization, vs. the 
number of training steps. Batch Normalization helps the 
network train faster and achieve higher accuracy. (b, 
c) The evolution of input distributions to a typical sig- 
moid, over the course of training, shown as {15, 50, 85}th 
percentiles. Batch Normalization makes the distribution 
more stable and reduces the internal covariate shift. 


3 fully-connected hidden layers with 100 activations each. 
Each hidden layer computes y = g(Wu-+b) with sigmoid 
nonlinearity, and the weights W initialized to small ran- 
dom Gaussian values. The last hidden layer is followed 
by a fully-connected layer with 10 activations (one per 
class) and cross-entropy loss. We trained the network for 
50000 steps, with 60 examples per mini-batch. We added 
Batch Normalization to each hidden layer of the network, 
as in Sec. We were interested in the comparison be- 
tween the baseline and batch-normalized networks, rather 
than achieving the state of the art performance on MNIST 
(which the described architecture does not). 

Figure [Tla) shows the fraction of correct predictions 
by the two networks on held-out test data, as training 
progresses. The batch-normalized network enjoys the 
higher test accuracy. To investigate why, we studied in- 
puts to the sigmoid, in the original network N and batch- 
normalized network NE, (Alg.[2) over the course of train- 
ing. In Fig.[T[b,c) we show, for one typical activation from 
the last hidden layer of each network, how its distribu- 
tion evolves. The distributions in the original network 
change significantly over time, both in their mean and 
the variance, which complicates the training of the sub- 
sequent layers. In contrast, the distributions in the batch- 
normalized network are much more stable as training pro- 
gresses, which aids the training. 


4.2 ImageNet classification 


We applied Batch Normalization to a new variant of the 
Inception network (Szegedy et al 2014) trained on the 
ImageNet classification task (Russakovsky et al. 2014). 
The network has a large number of convolutional and 
pooling layers, with a softmax layer to predict the image 
class, out of 1000 possibilities. Convolutional layers use 
ReLU as the nonlinearity. The main difference to the net- 
work described in is that the 5 x 5 
convolutional layers are replaced by two consecutive lay- 
ers of 3 x 3 convolutions with up to 128 filters. The net- 
work contains 13.6 - 10% parameters, and, other than the 
top softmax layer, has no fully-connected layers. More 
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details are given in the Appendix. We refer to this model 
as Inception in the rest of the text. The model was trained 
using a version of Stochastic Gradient Descent with mo- 


mentum (Sutskever et al}|2013), using the mini-batch size 
of 32. The training was performed using a large-scale, dis- 
tributed architecture (similar to ETI E All 
networks are evaluated as training progresses by comput- 
ing the validation accuracy @1, i.e. the probability of 
predicting the correct label out of 1000 possibilities, on 
a held-out set, using a single crop per image. 

In our experiments, we evaluated several modifications 
of Inception with Batch Normalization. In all cases, Batch 
Normalization was applied to the input of each nonlinear- 
ity, in a convolutional way, as described in section 3.2] 
while keeping the rest of the architecture constant. 


4.2.1 Accelerating BN Networks 


Simply adding Batch Normalization to a network does not 
take full advantage of our method. To do so, we further 
changed the network and its training parameters, as fol- 
lows: 

Increase learning rate. In a batch-normalized model, 
we have been able to achieve a training speedup from 
higher learning rates, with no ill side effects (Sec.[3.3). 


Remove Dropout. As described in Sec.B.4] Batch Nor- 
malization fulfills some of the same goals as Dropout. Re- 
moving Dropout from Modified BN-Inception speeds up 
training, without increasing overfitting. 

Reduce the Lz weight regularization. While in Incep- 
tion an Lə loss on the model parameters controls overfit- 
ting, in Modified BN-Inception the weight of this loss is 
reduced by a factor of 5. We find that this improves the 
accuracy on the held-out validation data. 

Accelerate the learning rate decay. In training Incep- 
tion, learning rate was decayed exponentially. Because 
our network trains faster than Inception, we lower the 
learning rate 6 times faster. 


Remove Local Response Normalization While Incep- 
tion and other networks 2014) benefit 
from it, we found that with Batch Normalization it is not 
necessary. 

Shuffle training examples more thoroughly. We enabled 
within-shard shuffling of the training data, which prevents 
the same examples from always appearing in a mini-batch 
together. This led to about 1% improvements in the val- 
idation accuracy, which is consistent with the view of 
Batch Normalization as a regularizer (Sec. (3.4): the ran- 
domization inherent in our method should be most bene- 
ficial when it affects an example differently each time it is 
seen. 

Reduce the photometric distortions. Because batch- 
normalized networks train faster and observe each train- 
ing example fewer times, we let the trainer focus on more 
“real” images by distorting them less. 
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Figure 2: Single crop validation accuracy of Inception 
and its batch-normalized variants, vs. the number of 
training steps. 


4.2.2 Single-Network Classification 


We evaluated the following networks, all trained on the 
LSVRC2012 training data, and tested on the validation 
data: 

Inception: the network described at the beginning of 
Section[4.2] trained with the initial learning rate of 0.0015. 

BN-Baseline: Same as Inception with Batch Normal- 
ization before each nonlinearity. 

BN-x5: Inception with Batch Normalization and the 
modifications in Sec. The initial learning rate was 
increased by a factor of 5, to 0.0075. The same learning 
rate increase with original Inception caused the model pa- 
rameters to reach machine infinity. 

BN-x30: Like BN-x5, but with the initial learning rate 
0.045 (30 times that of Inception). 

BN-x5-Sigmoid: Like BN-x5, but with sigmoid non- 
linearity g(t) = mee instead of ReLU. We also at- 
tempted to train the original Inception with sigmoid, but 
the model remained at the accuracy equivalent to chance. 

In Figure [2| we show the validation accuracy of the 
networks, as a function of the number of training steps. 
Inception reached the accuracy of 72.2% after 31 - 10% 
training steps. The Figure [8] shows, for each network, 
the number of training steps required to reach the same 
72.2% accuracy, as well as the maximum validation accu- 
racy reached by the network and the number of steps to 
reach it. 

By only using Batch Normalization (BN-Baseline), we 
match the accuracy of Inception in less than half the num- 
ber of training steps. By applying the modifications in 
Sec. we significantly increase the training speed of 
the network. BN-x3 needs 14 times fewer steps than In- 
ception to reach the 72.2% accuracy. Interestingly, in- 
creasing the learning rate further (BN-x30) causes the 
model to train somewhat slower initially, but allows it to 
reach a higher final accuracy. It reaches 74.8% after 6-10° 
steps, i.e. 5 times fewer steps than required by Inception 
to reach 72.2%. 

We also verified that the reduction in internal covari- 
ate shift allows deep networks with Batch Normalization 
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Model Steps to 72.2% Max accuracy 
Inception 31.0 -10 72.2% 
BN-Baseline 13.3 - 10% 72.7% 
BN-x5 2.1 - 10° 73.0% 
BN-x30 2.7 - 10° 74.8% 
BN-x5-Sigmoid 69.8% 





Figure 3: For Inception and the batch-normalized 
variants, the number of training steps required to 
reach the maximum accuracy of Inception (72.2%), 
and the maximum accuracy achieved by the net- 
work. 


to be trained when sigmoid is used as the nonlinearity, 
despite the well-known difficulty of training such net- 
works. Indeed, BN-x5-Sigmoid achieves the accuracy of 
69.8%. Without Batch Normalization, Inception with sig- 
moid never achieves better than 1/1000 accuracy. 


4.2.3 Ensemble Classification 


The current reported best results on the ImageNet Large 
Scale Visual Recognition Competition are reached by the 
Deep Image ensemble of traditional models (Wu et al. 
and the ensemble model of 2015). The 
latter reports the top-5 error of 4.94%, as evaluated by the 
ILSVRC server. Here we report a top-5 validation error of 
4.9%, and test error of 4.82% (according to the ILSVRC 
server). This improves upon the previous best result, and 
exceeds the estimated accuracy of human raters according 
to (Russakovsky et al! [2014). 

For our ensemble, we used 6 networks. Each was based 
on BN-x30, modified via some of the following: increased 
initial weights in the convolutional layers; using Dropout 
(with the Dropout probability of 5% or 10%, vs. 40% 
for the original Inception); and using non-convolutional, 
per-activation Batch Normalization with last hidden lay- 
ers of the model. Each network achieved its maximum 
accuracy after about 6 - 10% training steps. The ensemble 
prediction was based on the arithmetic average of class 
probabilities predicted by the constituent networks. The 
details of ensemble and multicrop inference are similar to 
(Szegedy et al! (2014). 

We demonstrate in Fig. [4]that batch normalization al- 
lows us to set new state-of-the-art by a healthy margin on 
the ImageNet classification challenge benchmarks. 


5 Conclusion 


We have presented a novel mechanism for dramatically 
accelerating the training of deep networks. It is based on 
the premise that covariate shift, which is known to com- 
plicate the training of machine learning systems, also ap- 
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Model Resolution Crops Models Top-l error Top-5 error 
GoogLeNet ensemble 224 144 7 - 6.67% 
Deep Image low-res 256 - 1 - 7.96% 
Deep Image high-res 512 - 1 24.88 7.42% 
Deep Image ensemble variable - - - 5.98% 
BN-Inception single crop 224 1 1 25.2% 7.82% 
BN-Inception multicrop 224 144 1 21.99% 5.82% 
BN-Inception ensemble 224 144 6 20.1% 4.9%* 


Figure 4: Batch-Normalized Inception comparison with previous state of the art on the provided validation set com- 
prising 50000 images. *BN-Inception ensemble has reached 4.82% top-5 error on the 100000 images of the test set of 


the ImageNet as reported by the test server. 


plies to sub-networks and layers, and removing it from 
internal activations of the network may aid in training. 
Our proposed method draws its power from normalizing 
activations, and from incorporating this normalization in 
the network architecture itself. This ensures that the nor- 
malization is appropriately handled by any optimization 
method that is being used to train the network. To en- 
able stochastic optimization methods commonly used in 
deep network training, we perform the normalization for 
each mini-batch, and backpropagate the gradients through 
the normalization parameters. Batch Normalization adds 
only two extra parameters per activation, and in doing so 
preserves the representation ability of the network. We 
presented an algorithm for constructing, training, and per- 
forming inference with batch-normalized networks. The 
resulting networks can be trained with saturating nonlin- 
earities, are more tolerant to increased training rates, and 
often do not require Dropout for regularization. 


Merely adding Batch Normalization to a state-of-the- 
art image classification model yields a substantial speedup 
in training. By further increasing the learning rates, re- 
moving Dropout, and applying other modifications af- 
forded by Batch Normalization, we reach the previous 
state of the art with only a small fraction of training steps 
— and then beat the state of the art in single-network image 
classification. Furthermore, by combining multiple mod- 
els trained with Batch Normalization, we perform better 
than the best known system on ImageNet, by a significant 
margin. 


Interestingly, our method bears similarity to the stan- 
dardization layer of ¿el though 
the two methods stem from very different goals, and per- 
form different tasks. The goal of Batch Normalization 
is to achieve a stable distribution of activation values 
throughout training, and in our experiments we apply it 
before the nonlinearity since that is where matching the 
first and second moments is more likely to result in a 
stable distribution. On the contrary, 
apply the standardization layer to the output of the 
nonlinearity, which results in sparser activations. In our 
large-scale image classification experiments, we have not 
observed the nonlinearity inputs to be sparse, neither with 
nor without Batch Normalization. Other notable differ- 
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entiating characteristics of Batch Normalization include 
the learned scale and shift that allow the BN transform 
to represent identity (the standardization layer did not re- 
quire this since it was followed by the learned linear trans- 
form that, conceptually, absorbs the necessary scale and 
shift), handling of convolutional layers, deterministic in- 
ference that does not depend on the mini-batch, and batch- 
normalizing each convolutional layer in the network. 

In this work, we have not explored the full range of 
possibilities that Batch Normalization potentially enables. 
Our future work includes applications of our method to 
Recurrent Neural Networks (Pascanu et al. (2013), where 
the internal covariate shift and the vanishing or exploding 
gradients may be especially severe, and which would al- 
low us to more thoroughly test the hypothesis that normal- 
ization improves gradient propagation (Sec.B.3). We plan 
to investigate whether Batch Normalization can help with 
domain adaptation, in its traditional sense — i.e. whether 
the normalization performed by the network would al- 
low it to more easily generalize to new data distribu- 
tions, perhaps with just a recomputation of the population 
means and variances (Alg.[2). Finally, we believe that fur- 
ther theoretical analysis of the algorithm would allow still 
more improvements and applications. 
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Appendix 


Variant of the Inception Model Used 


Figure [5] documents the changes that were performed 
compared to the architecture with respect to the 
GoogleNet archictecture. For the interpretation of this 
table, please consult 2014). The notable 
architecture changes compared to the GoogLeNet model 
include: 


e The 5x5 convolutional layers are replaced by two 
consecutive 3x3 convolutional layers. This in- 
creases the maximum depth of the network by 9 
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weight layers. Also it increases the number of pa- 
rameters by 25% and the computational cost is in- 
creased by about 30%. 


e The number 28 x28 inception modules is increased 
from 2 to 3. 


e Inside the modules, sometimes average, sometimes 
maximum-pooling is employed. This is indicated in 
the entries corresponding to the pooling layers of the 
table. 


e There are no across the board pooling layers be- 
tween any two Inception modules, but stride-2 con- 
volution/pooling layers are employed before the fil- 
ter concatenation in the modules 3c, 4e. 


Our model employed separable convolution with depth 
multiplier 8 on the first convolutional layer. This reduces 
the computational cost while increasing the memory con- 
sumption at training time. 
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patch size, output ES x double #3 x double a 
[convolution® | 7x7/2 [nxa O OO O | T T i 
Pmaxpool | 3x52 | sexsex | 0 [| [| [| TOO | 7 
Feonvotution | 3x31 | 56x56x192 | 1 | [6 oao [| |] 
Pmaxpool | 3x3/2 | 28x28x192 | 0 [| P [| [| [| | 7 
[inception Ga [2x3 apa pa] ea | 9 [| aves | 
[inception (3b) |__| 28x28x320 | 3 | 64 | o [| 9 | 6 | 9 | averós |] 
[inception (Ge) | stride 2 | 28x28x576 | 3 | 0 | ms | 160 | 6% | 96 | max + pass through | 
[mepo] | maese | 3 | 24 4 | 9 | 9 | vg | 
[inception (49) ee | 3 | 192 | 96 [| 18 | 96 | 198 | aw+98 | 
[inception ey | | aia | 3 | 160 | 18 | 160 | 98 | 160 vg | 
[inception (4d) | | matas | 3 | 9 | 18 | 192 | 160 | 192 avg | 
[mepo] | oaoa | 3 | 352 | 192 | 320 | 160 | 24 |  ave+ms | 
[mepi] | oaoa | 3 | 352 | 192 | 320 | 192 > | 24 | max +128 
[avg pool | _7x7/i_|_ixixioa | o [| | | | | | 7] 





Figure 5: Inception architecture 
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